Hydrologic Information Systems: Advancing Cyberinfrastructure for Environmental Observatories

Recently, community initiatives have emerged for the establishment of large-scale environmental observatories. Cyberinfrastructure is the backbone upon which these observatories will be built, and scientists\u27 ability to access and use the data collected within observatories to address research questions will depend on the successful implementation of cyberinfrastructure. The research described in this dissertation advances the cyberinfrastructure available for supporting environmental observatories. This has been accomplished through both development of new cyberinfrastructure components as well as through the demonstration and application of existing tools, with a specific focus on point observations data. The cyberinfrastructure that was developed and deployed to support collection, management, analysis, and publication of data generated by an environmental sensor network in the Little Bear River environmental observatory test bed is described, as is the sensor network design and deployment. Results of several analyses that demonstrate how high-frequency data enable identification of trends and analysis of physical, chemical, and biological behavior that would be impossible using traditional, low-frequency monitoring data are presented. This dissertation also illustrates how the cyberinfrastructure components demonstrated in the Little Bear River test bed have been integrated into a data publication system that is now supporting a nationwide network of 11 environmental observatory test bed sites, as well as other research sites within and outside of the United States. Enhancements to the infrastructure for research and education that are enabled by this research are impacting a diverse community, including the national community of researchers involved with prospective Water and Environmental Research Systems (WATERS) Network environmental observatories as well as other observatory efforts, research watersheds, and test beds. The results of this research provide insight into and potential solutions for some of the bottlenecks associated with design and implementation of cyberinfrastructure for observatory support

ODM Tools Python: Open Source Software For Managing Continuous Sensor Data

Hydrologic and water quality data is being collected at high frequencies, for extended durations, and with spatial distributions that require infrastructure for data storage and management. The Observations Data Model (ODM), which is part of the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) Hydrologic Information System (HIS), was developed as a framework in which to organize, store, and describe point observations data. In this paper we describe ODM Tools Python, which is an open source software application that allows ODM users to query and export, visualize, and edit data stored in an ODM database. Previous versions of ODM Tools included functionality to export data series and associated metadata, plot and summarize single data series, generate derivative data series, and edit data series using a set of simple tools. We have developed a new version of ODM Tools in Python that adds a modernized graphical user interface, multiple platform support (Windows, Linux, and Mac), multiple database support (Microsoft SQL Server and MySQL), and support for automated scripting of quality control edits performed on data series through an integrated Python script editor and console. Scripting records the corrections and adjustments made to data series in the quality control process, ensuring that the steps are traceable and reproducible. Additional improvements to ODM Tools Python include customizable queries for data selection and export, the ability to plot multiple data series simultaneously with various plot types, and user-defined functions for data series editing and derivation

ODM Tools Python: Open Source Software For Managing Continuous Sensor Data

Hydrologic and water quality data is being collected at high frequencies, for extended durations, and with spatial distributions that require infrastructure for data storage and management. The Observations Data Model (ODM), which is part of the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) Hydrologic Information System (HIS), was developed as a framework in which to organize, store, and describe point observations data. In this paper we describe ODM Tools Python, which is an open source software application that allows ODM users to query and export, visualize, and edit data stored in an ODM database. Previous versions of ODM Tools included functionality to export data series and associated metadata, plot and summarize single data series, generate derivative data series, and edit data series using a set of simple tools. We have developed a new version of ODM Tools in Python that adds a modernized graphical user interface, multiple platform support (Windows, Linux, and Mac), multiple database support (Microsoft SQL Server and MySQL), and support for automated scripting of quality control edits performed on data series through an integrated Python script editor and console. Scripting records the corrections and adjustments made to data series in the quality control process, ensuring that the steps are traceable and reproducible. Additional improvements to ODM Tools Python include customizable queries for data selection and export, the ability to plot multiple data series simultaneously with various plot types, and user-defined functions for data series editing and derivation

Assessing Subjectivity in Environmental Sensor Data Post Processing via a Controlled Experiment

Collection of high resolution, in situ data using environmental sensors is common in hydrology and other environmental science domains. Sensors are subject to drift, fouling, and other factors that can affect the quality of the measurements and their subsequent use for scientific analyses. The process by which sensor data are reviewed to verify validity often requires making edits in post processing to generate approved datasets. This quality control process involves decisions by technicians, data managers, or data users on how to handle problematic data. In this study, an experiment was designed and conducted where multiple participants performed quality control post processing on the same datasets using consistent guidelines and tools to assess the effect of individual technician on the resulting datasets. The effect of technician experience and training was also assessed by conducting the same procedures with a group of novices unfamiliar with the data and compared results to those generated by a group of experienced technicians. Results showed greater variability between outcomes for experienced participants, which we attribute to novice participants\u27 reluctance to implement unfamiliar procedures that change data. The greatest variability between participants\u27 results was associated with calibration events for which users selected different methods and values by which to shift results. These corrections resulted in variability exceeding the range of manufacturer-reported sensor accuracy. To reduce quality control subjectivity and variability, we recommend that monitoring networks establish detailed quality control guidelines and consider a collaborative approach to quality control in which multiple technicians evaluate datasets prior to publication

A Low-Cost, Open Source Monitoring System for Collecting High Temporal Resolution Water Use Data on Magnetically Driven Residential Water Meters

We present a low-cost (≈$150) monitoring system for collecting high temporal resolution residential water use data without disrupting the operation of commonly available water meters. This system was designed for installation on top of analog, magnetically driven, positive displacement, residential water meters and can collect data at a variable time resolution interval. The system couples an Arduino Pro microcontroller board, a datalogging shield customized for this specific application, and a magnetometer sensor. The system was developed and calibrated at the Utah Water Research Laboratory and was deployed for testing on five single family residences in Logan and Providence, Utah, for a period of over 1 month. Battery life for the device was estimated to be over 5 weeks with continuous data collection at a 4 s time interval. Data collected using this system, under ideal installation conditions, was within 2% of the volume recorded by the register of the meter on which they were installed. Results from field deployments are presented to demonstrate the accuracy, functionality, and applicability of the system. Results indicate that the device is capable of collecting data at a temporal resolution sufficient for identifying individual water use events and analyzing water use at coarser temporal resolutions. This system is of special interest for water end use studies, future projections of residential water use, water infrastructure design, and for advancing our understanding of water use timing and behavior. The system’s hardware design and software are open source, are available for potential reuse, and can be customized for specific research needs

Advancing the Open Modeling Interface (OpenMI) for Integrated Water Resources Modeling

The use of existing component-based modeling frameworks for integrated water resources modeling is currently hampered for some important use cases because they lack support for commonly used, topology-aware, spatiotemporal data structures. Additionally, existing frameworks are often accompanied by large software stacks with steep learning curves. Others lack specifications for deploying them on high performance, heterogeneous computing (HPC) infrastructure. This puts their use beyond the reach of many water resources modelers. In this paper, we describe new advances in component-based modeling using a framework called HydroCouple. This framework largely adopts the Open Modeling Interface (OpenMI) 2.0 interface definitions but demonstrates important advances for water resources modeling. HydroCouple explicitly defines standard and widely used geospatial data formats and provides interface definitions to support simulations on HPC infrastructure. In this paper, we illustrate how these advances can be used to develop efficient model components through a coupled urban stormwater modeling exercise

Advancing Cyberinfrastructure to support high resolution water resources modeling (Invited)

Addressing the problem of how the availability and quality of water resources at large scales are sensitive to climate variability, watershed alterations and management activities requires computational resources that combine data from multiple sources and support integrated modeling. Related cyberinfrastructure challenges include: 1) how can we best structure data and computer models to address this scientific problem through the use of high-performance and data-intensive computing, and 2) how can we do this in a way that discipline scientists without extensive computational and algorithmic knowledge and experience can take advantage of advances in cyberinfrastructure? This presentation will describe a new system called CI-WATER that is being developed to address these challenges and advance high resolution water resources modeling in the Western U.S. We are building on existing tools that enable collaboration to develop model and data interfaces that link integrated system models running within an HPC environment to multiple data sources. Our goal is to enhance the use of computational simulation and data-intensive modeling to better understand water resources. Addressing water resource problems in the Western U.S. requires simulation of natural and engineered systems, as well as representation of legal (water rights) and institutional constraints alongside the representation of physical processes. We are establishing data services to represent the engineered infrastructure and legal and institutional systems in a way that they can be used with high resolution multi-physics watershed modeling at high spatial resolution. These services will enable incorporation of location-specific information on water management infrastructure and systems into the assessment of regional water availability in the face of growing demands, uncertain future meteorological forcings, and existing prior-appropriations water rights. This presentation will discuss the informatics challenges involved with data management and easy-to-use access to high performance computing being tackled in this project